Methods for identifying audio or video content

ABSTRACT

The present disclosure relates to analyzing user uploaded visual or audio content. One method comprises: generating fingerprint data from user uploaded visual or audio content, the user uploaded visual or audio content having been uploaded by a user to a first party&#39;s online video service for distribution from the first party&#39;s online video service to the public; identifying plural items of reference content, from a universe of reference content, that are a possible match of the user uploaded visual or audio content, said identifying being based on the fingerprint data and being performed by a computer system configured to perform such act; assessing a score for each of the identified plural items of reference content, each score indicating a likely correspondence of the user uploaded visual or audio content with a respective item of reference content; for items in the plural items of reference content that fall within a predetermined score range, queuing for review by a second party, different than the first party, the user uploaded visual or audio content item and data associated with the items in the plural items of reference content that fall within a predetermined score range; receiving assessment data from the second party, the assessment data indicating a result of a subjective comparison of the user uploaded visual or audio content and the items in the plural items of reference content that fall within a predetermined score range; using the assessment data and a score assessed in said act of assessing in concluding that at least one item in the identified plural items of reference content corresponds with the user uploaded visual or audio content; and limiting distribution of the user uploaded visual or audio content from the first party&#39;s online video service to the public based on the concluding. Of course other methods and combinations are disclosed as well.

RELATED APPLICATION DATA

This application is a continuation of U.S. patent application Ser. No.13/714,930, filed Dec. 14, 2012 (now U.S. Pat. No. 8,458,482) which is acontinuation of U.S. patent application Ser. No. 12/114,612, filed May2, 2008 (now U.S. Pat. No. 8,341,412), which is a division of U.S.patent application Ser. No. 11/613,891, filed Dec. 20, 2006 (publishedas US 2007-0162761 A1), which claims priority to provisional application60/753,652, filed Dec. 23, 2005. Each of the above patent documents ishereby incorporated herein by reference in its entirety.

Some of the subject matter herein is related to that in various of theassignee's other patent applications, including Ser. No. 10/723,240,filed Nov. 26, 2003 (published as US20040213437); Ser. No. 10/979,770,filed Nov. 1, 2004 (now U.S. Pat. No. 7,314,162); and Ser. No.11/132,724, filed May 18, 2005 (published as US20050288952).

TECHNICAL FIELD

The technology detailed herein spans a range of subject matter,including identifying audio/video entertainment content.

BACKGROUND

Certain implementations of the present technology make use of Amazon'sMechanical Turk service. Amazon's Turk web site explains:

-   -   Amazon Mechanical Turk provides a web services API for computers        to integrate Artificial Intelligence directly into their        processing by making requests of humans. Developers use the        Amazon Mechanical Turk web services API to submit tasks to the        Amazon Mechanical Turk web site, approve completed tasks, and        incorporate the answers into their software applications. To the        application, the transaction looks very much like any remote        procedure call—the application sends the request, and the        service returns the results. In reality, a network of humans        fuels this Artificial Intelligence by coming to the web site,        searching for and completing tasks, and receiving payment for        their work.    -   All software developers need to do is write normal code. The        pseudo code below illustrates how simple this can be.

read (photo); photoContainsHuman = callMechanicalTurk(photo); if(photoContainsHuman == TRUE){  acceptPhoto; } else {  rejectPhoto; }

More information about Amazon's Mechanical Turk service is provided inthe attached Appendix A (Amazon Mechanical Turk Developer Guide, 2006,165 pp., API Version 10-31-2006).

The Mechanical Turk service may be regarded as a structuredimplementation of a technology commonly termed “crowdsourcing”—employinga group of outsiders to perform a task. Wikipedia explains:

-   -   “Crowdsourcing” is a neologism for a business model that depends        on work being done outside the traditional company walls: while        outsourcing is typically performed by lower paid professionals,        crowdsourcing relies on a combination of volunteers and low-paid        amateurs who use their spare time to create content, solve        problems, or even do corporate R&D. The term was coined by Wired        magazine writer Jeff Howe and editor Mark Robinson in June 2006.        Crowds targeted for crowdsourcing include garage scientists,        amateur videographers, freelancers, photo enthusiasts, data        companies, writers, smart mobs and the electronic herd.

Overview

-   -   While not a new idea, crowdsourcing is becoming mainstream. Open        source projects are a form of crowdsourcing that has existed for        years. People who may not know one another work together online        to create complex software such as the Linux kernel, and the        Firefox browser. In recent years internet technology has evolved        to allow non-technical people to participate in online projects.        Just as important, crowdsourcing presumes that a large number of        enthusiasts can outperform a small group of experienced        professionals.

Advantages

-   -   The main advantages of crowdsourcing is that innovative ideas        can be explored at relatively little cost. Furthermore, it also        helps reduce costs. For example if customers reject a particular        design, it can easily be scrapped. Though disappointing, this is        far less expensive than developing high volumes of a product        that no one wants. Crowdsourcing is also related to terms like        Collective Customer Commitment (CCC) and Mass Customisation.        Collective Customer Commitment (CCC) involves integrating        customers into innovation processes. It helps companies exploit        a pool of talent and ideas and it also helps firms avoid product        flops. Mass Customisation is somewhat similar to collective        customer commitment; however, it also helps companies avoid        making risky decisions about what components to prefabricate and        thus avoids spending for products which may not be marketable        later.

Types of Crowdsourced Work

-   -   Steve Jackson Games maintains a network of MIB (Men In Black),        who perform secondary jobs (mostly product representation) in        exchange for free product. They run publicly or semi-publicly        announced play-tests of all their major books and game systems,        in exchange for credit and product. They maintain an active user        community online, and have done so since the days of BBSes.    -   Procter & Gamble employs more than 9000 scientists and        researchers in corporate R&D and still have many problems they        can't solve. They now post these on a website called        InnoCentive, offering large cash rewards to more than 90,000        ‘solvers’ who make up this network of backyard scientists. P&G        also works with NineSigma, YourEncore and Yet2.    -   Amazon Mechanical Turk co-ordinates the use of human        intelligence to perform tasks which computers are unable to do.    -   YRUHRN used Amazon Mechanical Turk and other means of        crowdsourcing to compile content for a book published just 30        days after the project was started.    -   iStockphoto is a website with over 22,000 amateur photographers        who upload and distribute stock photographs. Because it does not        have the same margins as a professional outfit like Getty Images        it is able to sell photos for a low price. It was recently        purchased by Getty Images.    -   Cambrian House applies a crowdsourcing model to identify and        develop profitable software ideas. Using a simple voting model,        they attempt to find sticky software ideas that can be developed        using a combination of internal and crowdsourced skills and        effort.    -   A Swarm of Angels is a project to utilize a swarm of subscribers        (Angels) to help fund, make, contribute, and distribute, a £1        million feature film using the Internet and all digital        technologies. It aims to recruit earlier development community        members with the right expertise into paid project members, film        crew, and production staff.    -   The Goldcorp Challenge is an example of how a traditional        company in the mining industry used a crowdsource to identify        likely veins of gold on its Red Lake Property. It was won by        Fractal Graphics and Taylor-Wall and Associates of Australia but        more importantly identified 110 drilling targets, 50% of which        were new to the company.    -   CafePress and Zazzle, customized products marketplaces for        consumers to create apparel, posters, cards, stamps, and other        products.    -   Marketocracy, to isolating top stock market investors around the        world in head to head competition so they can run real mutual        funds around these soon-to-be-discovered investment super-stars.    -   Threadless, an internet-based clothing retailer that sells        t-shirts which have been designed by and rated by its users.    -   Public Insight Journalism, A project at American Public Media to        cover the news by tapping the collective and specific        intelligence of the public. Gets the newsroom beyond the usual        sources, uncovers unexpected expertise, stories and new angles.

EXTERNAL LINKS AND REFERENCES

-   The Rise of Crowdsourcing, Wired June 2006.-   Crowdsourcing: Consumers as Creators, BusinessWeek July 2006.

SUMMARY

The following text presents a simplified, incomplete summary in order toprovide an orientation to certain aspects of the disclosed subjectmatter. This Summary is not an extensive overview. It is not intended toidentify key/critical elements or to delineate the scope of the claimedsubject matter. Its sole purpose is to present some concepts in asimplified form as a prelude to the more detailed description thatfollows.

In accordance with certain embodiments of the present technology,Amazon's Mechanical Turk system, or similar crowdsourcing arrangements,are employed to match a first item of visual or audio entertainmentcontent to a counterpart in a universe of such items.

For example, consider a user social networking site such as YouTube (nowGoogle) that distributes “user generated content” (e.g., video files),and employs fingerprinting to recognize media content that should not bedistributed. The site may check a video file at the time of itsuploading with a fingerprint recognition system (e.g., of the sortoffered by Audible Magic, or Gracenote). If no clear match isidentified, the video may be indexed and stored on YouTube's servers,available for public downloading. Meanwhile, the content can be queuedfor review by one or more crowdsource reviewers. They may recognize itas a clip from the old TV sitcom “I Love Lucy”—perhaps digitally rotated3 degrees to avoid fingerprint detection. This tentative identificationis returned to YouTube from the API call. YouTube can check thereturning metadata against a title list of works that should not bedistributed (e.g., per the request of copyright owners), and maydiscover that “I Love Lucy” clips should not be distributed. It can thenremove the content from public distribution. Additionally, thefingerprint database can be updated with the fingerprint of the rotatedversion of the I Love Lucy clip, allowing it to be immediatelyrecognized the next time it is encountered.

The foregoing and other examples, features and advantages of the presenttechnology will be more apparent from the following DetailedDescription.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing some components of an example computersystem.

FIG. 2 is a flow diagram showing various acts accomplished throughexecution of code by a processor.

FIG. 3 is a diagram showing various additional acts accomplished throughexecution of code by a processor.

FIG. 4 is a flow diagram showing other various acts accomplished throughexecution of code by a processor.

DETAILED DESCRIPTION

One use of the Mechanical Turk service (and similar crowdsourcingtechnologies) is in connection with computationally difficult tasks,such as identification of audio, video and imagery content. These tasksare sometimes addressed by so-called “fingerprint” technology, whichseeks to generate a “robust hash” of content (e.g., distilling a digitalfile of the content down to perceptually relevant features), and thencompare the thus-obtained fingerprint against a database of referencefingerprints computed from known pieces of content, to identify a “best”match. Such technology is detailed, e.g., in Haitsma, et al, “A HighlyRobust Audio Fingerprinting System,” Proc. Intl Conf on MusicInformation Retrieval, 2002; Cano et al, “A Review of AudioFingerprinting,” Journal of VLSI Signal Processing, 41, 271, 272, 2005;Kalker et al, “Robust Identification of Audio Using Watermarking andFingerprinting,” in Multimedia Security Handbook, CRC Press, 2005, andin patent documents WO02/065782, US20060075237, US20050259819, andUS20050141707.

A related technology is facial recognition—matching an unknown face to areference database of facial images. Again, each of the faces isdistilled down to a characteristic set of features, and a match issought between an unknown feature set, and feature sets corresponding toreference images. (The feature set may comprise eigenvectors or shapeprimitives.) Patent documents particularly concerned with suchtechnology include US20020031253, U.S. Pat. No. 6,292,575, U.S. Pat. No.6,301,370, U.S. Pat. No. 6,430,306, U.S. Pat. No. 6,466,695, and U.S.Pat. No. 6,563,950.

These are examples of technology that relies on “fuzzy” matching. Thefingerprint derived from the unknown content often will not exactlymatch any of the reference fingerprints in the database. Thus, thedatabase must be searched not just for the identical contentfingerprint, but also for variants.

Expanding the search to include variants hugely complicates—andslows—the database search task. To make the search tractable, oneapproach is to prune the database—identifying excerpts thereof that arebelieved to be relatively likely to have a match, and limiting thesearch to those excerpts (or, similarly, identifying excerpts that arebelieved relatively unlikely to have a match, and not searching thoseexcerpts).

The database search may locate several reference fingerprints that aresimilar to the fingerprint of the unknown content. The identificationprocess then seeks to identify a “best” match, using various algorithms.

Such content identification systems can be improved by injecting a humaninto the process—by the Mechanical Turk service or similar systems.

In one particular arrangement, the content identification system makesan assessment of the results of its search, e.g., by a score. A score of100 may correspond to a perfect match between the unknown fingerprintand a reference fingerprint. Lower scores may correspond to successivelyless correspondence. (At some lower score, S_(x), (perhaps 60) thesystem may decide that there is no suitable match, and a “no-match”result is returned, with no identification made.)

Above some threshold score, S_(y), (perhaps 70) the system may besufficiently confident of the result that no human intervention isnecessary. At scores below S_(y), the system may make a call through theMechanical Turk service for assistance.

The Mechanical Turk can be presented the unknown content (or an excerptthereof), and some reference content, and asked to make a comparison.(The reference content may be stored in the fingerprint database, or maybe readily obtainable through use of a link stored in the referencedatabase.)

A single item of reference content can be provided for comparison withthe unknown content, or several items of reference content can beprovided. (Again, excerpts may be used instead of the complete contentobjects. Depending on the application, the content might be processedbefore sending to the crowdsource engine, e.g., removing metadata (suchas personally identifiable information: name, driver license number,etc.) that is printed on, or conveyed with, the file.)

The requested comparison can take different forms. The service can beasked simply whether two items appear to match. Or it can be asked toidentify the best of several possible matches (or indicate that noneappears to match). Or it can be asked to give a relative match score(e.g., 0-100) between the unknown content and one or more itemsreference content.

In many embodiments, a query is referred to several different humans(e.g., 2-50) through the Mechanical Turk service, and the returnedresults are examined for consensus on a particular answer. In somequeries (e.g., does Content A match Content B? Or is Content A a bettermatch to Content C?), a “vote” may be taken. A threshold of consensus(e.g., 51%, 75%, 90%, 100%) may be required in order for the serviceresponse to be given weight in the final analysis. Likewise, in queriesthat ask the humans to provide a subjective score, the scores returnedfrom plural such calls may be combined to yield a net result. (The highand/or low and/or outlier scores may be disregarded in computing the netresult; weighting can sometimes be employed, as noted below.)

As suggested, the data returned from the Mechanical Turk calls may serveas a biasing factor, e.g., pushing an algorithmically determined outputone way or another, to yield a final answer (e.g., a net score). Or thedata returned from the Mechanical Turk calls may be treated as adefinitive answer—with results from preceding processes disregarded.

Sometimes the database search may reveal several candidate matches, allwith comparable scores (which may be above the threshold S_(y)). Again,one or more calls to the Mechanical Turk service may be invoked todecide which match is the best, from a subjective human standpoint.

Sometimes the Mechanical Turk service can be invoked even in situationswhere the original confidence score is below the threshold, S_(x), whichis normally taken as indicating “no match.” Thus, the service can beemployed to effectively reduce this threshold—continuing to search forpotential matches when the rote database search does not yield anyresults that appear reliable.

The service can also be invoked to effect database pruning. For example,a database may be organized with several partitions (physical orlogical), each containing information of a different class. In a facialrecognition database, the data may be segregated by subject gender(i.e., male facial portraits, female facial portraits), and/or by age(15-40, 30-65, 55 and higher—data may sometimes be indexed in two ormore classifications), etc. In an image database, the data may besegregated by topical classification (e.g., portrait, sports, news,landscape). In an audio database, the data may be segregated by type(spoken word, music, other). Each classification, in turn, can befurther segregated (e.g., “music” may be divided into classical,country, rock, other). And these can be further segregated (e.g., “rock”may be classified by genre, such as soft rock, hard rock, Southern rock;by artist, e.g., Beatles, Rolling Stones, etc).

A call to the Mechanical Turk can be made, passing the unknown contentobject (or an excerpt thereof) to a human reviewer, soliciting advice onclassification. The human can indicate the apparent class to which theobject belongs (e.g., is this a male or female face? Is this musicclassical, country, rock, or other?). Or, the human can indicate one ormore classes to which the object does not belong.

With such human advice (which, again, may involve several humanreviewers, with a voting or scoring arrangement), the system can focusthe database search where a correct match—if any—is more likely to befound (or avoid searching in unproductive database excerpts). Thisfocusing can be done at different times. In one scenario it is doneafter a rote search is completed, in which the search results yieldmatches below the desired confidence level of S_(y). If the databasesearch space is thereafter restricted by application of human judgment,the search can be conducted again in the limited search space. A morethorough search can be undertaken in the indicated subset(s) of thedatabase. Since a smaller excerpt is being searched, a looser criteriafor a “match” might be employed, since the likelihood of false-positivematches is diminished. Thus, for example, the desired confidence levelS_(y) might be reduced from 70 to 65. Or the threshold S_(x) at which“no match” is concluded, may be reduced from 60 to 55. Alternatively,the focusing can be done before any rote searching is attempted.

The result of such a human-focused search may reveal one or morecandidate matches. The Mechanical Turk service may be called a secondtime, to vet the candidate matches—in the manner discussed above. Thisis one of several cases in which it may be desirable to cascadeMechanical Turk calls—the subsequent calls benefiting from the former.

In the example just-given, the first Mechanical Turk call aids inpruning the database for subsequent search. The second call aids inassessing the results of that subsequent search. In other arrangements,Mechanical Turk calls of the same sort can be cascaded.

For example, the Mechanical Turk first may be called to identify audioas music/speech/other. A second call may identify music (identified perthe first call) as classical/country/rock/other. A third call mayidentify rock (identified per the second call) as Beatles/RollingStones/etc. Here, again, by iterative calling of a crowdsourcingservice, a subjective judgment can be made that would be very difficultto achieve otherwise.

In some arrangements, human reviewers are pre-qualified as knowledgeablein a specific domain (e.g., relatively expert in recognizing Beatlesmusic). This qualification can be established by an online examination,which reviewers are invited to take to enable them to take on specifictasks (often at an increased rate of pay). Some queries may be routedonly to individuals that are pre-qualified in a particular knowledgedomain. In the cascaded example just given, for example, the third callmight be routed to one or more users with demonstrated expertise withthe Beatles (and, optionally, to one or more users with demonstratedexpertise with the Rolling Stones, etc). A positive identification ofthe unknown content as sounding like the Beatles would be given morerelative weight if coming from a human qualified in this knowledgedomain. (Such weighting may be taken into account when aggregatingresults from plural human reviewers. For example, consider an unknownaudio clip sent to six reviewers, two with expertise in the Beatles, twowith expertise in the Rolling Stones, and two with expertise in theGrateful Dead. Assume the Beatles experts identify it as Beatles music,the Rolling Stones experts identify it as Grateful Dead music, and theGrateful Dead experts identify it as Rolling Stones music. Despite thefact that there are tie votes, and despite the fact that no selectionearned a majority of the votes, the content identification service thatmade these calls and is provided with these results may logicallyconclude that the music is Beatles.)

Calls to the Mechanical Turk service may request the human to providemetadata relevant to any content reviewed. This can include supposedartist(s), genre, title, subject, date, etc. This information (which maybe ancillary to a main request, or may comprise the entirety of therequest) can be entered into a database. For example, it can be enteredinto a fingerprint database—in association with the content reviewed bythe human.

Desirably, data gleaned from Mechanical Turk calls are entered into thedatabase, and employed to enrich its data—and enrich information thatcan be later mined from the database. For example, if unknown content Xhas a fingerprint F_(x), and through the Mechanical Turk service it isdetermined that this content is a match to reference content Y, withfingerprint F_(y), then a corresponding notation can be added to thedatabase, so that a later query on fingerprint F_(x) (or close variantsthereof) will indicate a match to content Y. (E.g., a lookup tableinitially indexed with a hash of the fingerprint F_(x) will point to thedatabase record for content Y.)

Calls to outsourcing engines involve a time lag before results arereturned. The calling system can generally cope, or be adapted to cope,with such lags.

Consider a user social networking site such as YouTube (now Google) thatdistributes “user generated content” (e.g., video files), and employsfingerprinting to recognize media content that should not bedistributed. The site may check a video file at the time of itsuploading with a fingerprint recognition system (e.g., of the sortoffered by Audible Magic, or Gracenote). If no clear match isidentified, the video may be indexed and stored on YouTube's servers,available for public downloading. Meanwhile, the content can be queuedfor review by one our more crowdsource reviewers. They may recognize itas a clip from the old TV sitcom “I Love Lucy”—perhaps digitally rotated3 degrees to avoid fingerprint detection. This tentative identificationis returned to YouTube from the API call. YouTube can check thereturning metadata against a title list of works that should not bedistributed (e.g., per the request of copyright owners), and maydiscover that “I Love Lucy” clips should not be distributed. It can thenremove the content from public distribution. (This generally follows adouble-check of the identification by a YouTube employee.) Additionally,the fingerprint database can be updated with the fingerprint of therotated version of the I Love Lucy clip, allowing it to be immediatelyrecognized the next time it is encountered.

If the content is already being delivered to a user at the moment thedetermination is made (i.e., the determination that the content shouldnot be distributed publicly), then the delivery can be interrupted. Anexplanatory message can be provided to the user (e.g., a splash screenpresented at the interruption point in the video).

FIG. 1 shows an example computer system (10) including a processor (12)and a computer-readable readable storage medium (14) storing executablecode. The code when executed by the processor (12) can be configured toperform various acts. For example, FIG. 2 is a flow diagram showing someof these acts. FIG. 3 illustrates additional acts. FIG. 4 is a flowdiagram showing other example acts.

Rotating a video by a few degrees is one of several hacks that candefeat fingerprint identification. (It is axiomatic that introduction ofany new content protection technology draws hacker scrutiny. Familiarexamples include attacks against Macrovision protection for VHS tapes,and against CSS protection for packaged DVD discs.) If fingerprinting isemployed in content protection applications, such as in socialnetworking sites (as outlined above) or peer-to-peer networks, itsvulnerability to attack will eventually be determined and exploited.

Each fingerprinting algorithm has particular weaknesses that can beexploited by hackers to defeat same. An example will help illustrate.

A well known fingerprinting algorithm operates by repeatedly analyzingthe frequency content of a short excerpt of an audio track (e.g., 0.4seconds). The method determines the relative energy of this excerptwithin 33 narrow frequency bands that logarithmically span the range 300Hz-2000 Hz. A corresponding 32-bit identifier is then generated from theresulting data. In particular, a frequency band corresponds to a databit “1” if its energy level is larger than that of the band above, and a“0” if its energy level is lower. (A more complex arrangement can alsotake variations over time into account, outputting a “1” only if theimmediately preceding excerpt also met the same test, i.e., having aband energy greater than the band above.)

Such a 32 bit identifier is computed every hundredth of a second or so,for the immediately preceding 0.4 second excerpt of the audio track,resulting in a large number of “fingerprints.” This series ofcharacteristic fingerprints can be stored in a database entry associatedwith the track, or only a subset may be stored (e.g., every fourthfingerprint).

When an unknown track is encountered, the same calculation process isrepeated. The resulting set of data is then compared against dataearlier stored in the database to try and identify a match. (As noted,various strategies can be employed to speed the search over abrute-force search technique, which yields unacceptable search times.)

While the just-described technique is designed for audio identification,a similar arrangement can be used for video. Instead of energies inaudio subbands, the algorithm can use average luminances of blocks intowhich the image is divided as the key perceptual features. Again, afingerprint can be defined by determining whether the luminance in eachblock is larger or smaller than the luminance of the preceding block.

The just-reviewed fingerprinting algorithm is particularly detailed inthe Haitsma paper, referenced above. Four paragraphs from that paper,further detailing fingerprint extraction, are reproduced below:

Most fingerprint extraction algorithms are based on the followingapproach. First the audio signal is segmented into frames. For everyframe a set of features is computed. Preferably the features are chosensuch that they are invariant (at least to a certain degree) to signaldegradations. Features that have been proposed are well known audiofeatures such as Fourier coefficients, Mel Frequency CepstralCoefficients (MFFC), spectral flatness, sharpness, Linear PredictiveCoding (LPC) coefficients and others. Also derived quantities such asderivatives, means and variances of audio features are used. Generallythe extracted features are mapped into a more compact representation byusing classification algorithms, such as Hidden Markov Models, orquantization. The compact representation of a single frame is referredto as a sub-fingerprint. The global fingerprint procedure converts astream of audio into a stream of sub-fingerprints. One sub-fingerprintusually does not contain sufficient data to identify an audio clip. Thebasic unit that contains sufficient data to identify an audio clip (andtherefore determining the granularity) will be referred to as afingerprint-block.

The proposed fingerprint extraction scheme is based on this generalstreaming approach. It extracts 32-bit sub-fingerprints for everyinterval of 11.6 milliseconds. A fingerprint block consists of 256subsequent sub-fingerprints, corresponding to a granularity of only 3seconds. The audio signal is first segmented into overlapping frames.The overlapping frames have a length of 0.37 seconds and are weighted bya Hanning window with an overlap factor of 31/32. This strategy resultsin the extraction of one sub-fingerprint for every 11.6 milliseconds. Inthe worst-case scenario the frame boundaries used during identificationare 5.8 milliseconds off with respect to the boundaries used in thedatabase of pre-computed fingerprints. The large overlap assures thateven in this worst-case scenario the sub-fingerprints of the audio clipto be identified are still very similar to the sub-fingerprints of thesame clip in the database. Due to the large overlap subsequentsub-fingerprints have a large similarity and are slowly varying in time.

The most important perceptual audio features live in the frequencydomain. Therefore a spectral representation is computed by performing aFourier transform on every frame. Due to the sensitivity of the phase ofthe Fourier transform to different frame boundaries and the fact thatthe Human Auditory System (HAS) is relatively insensitive to phase, onlythe absolute value of the spectrum, i.e. the power spectral density, isretained.

In order to extract a 32-bit sub-fingerprint value for every frame, 33non-overlapping frequency bands are selected. These bands lie in therange from 300 Hz to 2000 Hz (the most relevant spectral range for theHAS) and have a logarithmic spacing. The logarithmic spacing is chosen,because it is known that the HAS operates on approximately logarithmicbands (the so-called Bark scale). Experimentally it was verified thatthe sign of energy differences (simultaneously along the time andfrequency axes) is a property that is very robust to many kinds ofprocessing.

Additional information on deriving fingerprints is provided in the Canopaper, A Review of Audio Fingerprinting, referenced above. Twoparagraphs from that reference—discussing linear transforms useful infingerprinting—follow:

The idea behind linear transforms is the projection of the set ofmeasurements to a new set of features. If the transform is suitablychosen, the redundancy is significantly reduced. There are optimaltransforms in the sense of information packing and decorrelationproperties, like Karhunen-Loeve (KL) or Singular Value Decomposition(SVD). These transforms, however, are problem dependent andcomputationally complex. For that reason, lower complexity transformsusing fixed basis vectors are common. Most CBID methods therefore usestandard transforms from time to frequency domain to facilitateefficient compression, noise removal and subsequent processing. Lourens,(for computational simplicity), and Kurth et al., (to model highlydistorted sequences, where the time-frequency analysis exhibitsdistortions), use power measures. The power can still be seen as asimplified time-frequency distribution, with only one frequency bin.

The most common transformation is the Discrete Fourier Transform (DFT).Some other transforms have been proposed: the Discrete Cosine Transform(DCT), the Haar Transform or the Walsh-Hadamard Transform. Richly et al.did a comparison of the DFT and the Walsh-Hadamard Transform thatrevealed that the DFT is generally less sensitive to shifting. TheModulated Complex Transform (MCLT) used by Mihcak et al. and also byBurges et al. exhibits approximate shift invariance properties.

While little has been written about attacks targeting fingerprintingsystems, a casual examination of possible attack scenarios revealsseveral possibilities. A true hacker will probably see many more. Foursimple approaches are discussed below.

Radio Loudness Profiling

The reader may be familiar with different loudness profiles selectableon car radios, e.g., Jazz, Talk, Rock, etc. Each applies a differentfrequency equalization profile to the audio, e.g., making bass noteslouder if the Rock setting is selected, and quieter if the Talk settingis selected, etc. The difference is often quite audible when switchingbetween different settings.

However, if the radio is simply turned on and tuned to differentstations, the listener is generally unaware of which loudness profile isbeing employed. That is, without the ability to switch between differentprofiles, the frequency equalization imposed by a particular loudnessprofile is typically not noticed by a listener. The different loudnessprofiles, however, yield different fingerprints.

For example, in the Rock setting, the 300 Hz energy in a particular 0.4second excerpt may be greater than the 318 Hz energy. However, in theTalk setting, the situation may be reversed. This change prompts achange in the leading bit of the fingerprint.

In practice, an attacker would probably apply loudness profiles morecomplex than those commonly available in car radios—increasing anddecreasing the loudness at many different frequency bands (e.g., 32different frequency bands). Significantly different fingerprints maythus be produced. Moreover, the loudness profile could change withtime—further distancing the resulting fingerprint from the referencevalues stored in a database.

Multiband Compression

Another process readily available to attackers is audio multibandcompression, a form of processing that is commonly employed bybroadcasters to increase the apparent loudness of their signal (mostespecially commercials). Such tools operate by reducing the dynamicrange of a soundtrack—increasing the loudness of quiet passages on aband-by-band basis, to thereby achieve a higher average signal level.Again, this processing of the audio changes its fingerprint, yet isgenerally not objectionable to the listeners.

Psychoacoustic Processing

The two examples given above are informal attacks—common signalprocessing techniques that yield, as side-effects, changes in audiofingerprints. Formal attacks—signal processing techniques that areoptimized for purposes of changing fingerprints—are numerous.

Some formal attacks are based on psychoacoustic masking. This is thephenomena by which, e.g., a loud sound at one instant (e.g., a drumbeat) obscures a listener's ability to perceive a quieter sound at alater instant. Or the phenomena by which a loud sound at one frequency(e.g., 338 Hz) obscures a listener's ability to perceive a quieter soundat a nearby frequency (e.g., 358 Hz) at the same instant. Research inthis field goes back decades. (Modern watermarking software employspsychoacoustic masking in an advantageous way, to help hide extra datain audio and video content.)

Hacking software, of course, can likewise examine a song'scharacteristics and identify the psychoacoustic masking opportunities itpresents. Such software can then automatically make slight alterationsin the song's frequency components in a way that a listener won't beable to note, yet in a way that will produce a different series ofcharacteristic fingerprints. The processed song will be audiblyindistinguishable from the original, but will not “match” any series offingerprints in the database.

Threshold Biasing

Another formal attack targets fingerprint bit determinations that arenear a threshold, and slightly adjusts the signal to swing the outcomethe other way. Consider an audio excerpt that has the followingrespective energy levels (on a scale of 0-99), in the frequency bandsindicated:

300 Hz 318 Hz 338 Hz 358 Hz 69 71 70 68

The algorithm detailed above would generate a fingerprint of {011 . . .} from this data (i.e., 69 is less than 71, so the first bit is ‘0’; 71is greater than 70, so the second bit is ‘1’; 70 is greater than 68, sothe third bit is ‘1’).

Seeing that the energy levels are somewhat close, an attacker tool couldslightly adjust the signal's spectral composition, so that the relativeenergy levels are as follows:

300 Hz 318 Hz 338 Hz 358 Hz [69] 70 [71] 69 70 68

Instead of {011 . . . }, the fingerprint is now {101 . . . }. Two of thethree illustrated fingerprint bits have been changed. Yet the change tothe audio excerpt is essentially inaudible.

Exploiting Database Pruning

Other fingerprint hacking vulnerabilities arise from shortcuts employedin the database searching strategy—seeking to prune large segments ofthe data from further searching. For example, the system outlined aboveconfines the large potential search space by assuming that there existsa 32 bit excerpt of the unknown song fingerprint that exactly matches(or matches with only one bit error) a 32 bit excerpt of fingerprintdata in the reference database. The system looks at successive 32 bitexcerpts from the unknown song fingerprint, and identifies all databasefingerprints that include an excerpt presenting a very close match(i.e., 0 or 1 errors). A list of candidate song fingerprints is therebyidentified that can be further checked to determine if any meets thelooser match criteria generally used. (To allow non-exact fingerprintmatches, the system generally allows up to 2047 bit errors in every 8192bit block of fingerprint data.)

The evident problem is: what if the correct “match” in the database hasno 32 bit excerpt that corresponds—with just 1 or 0 bit errors—to a 32bit excerpt from the unknown song? Such a correct match will never befound—it gets screened out at the outset.

A hacker familiar with the system's principles will see that everythinghinges on the assumption that a 32 bit string of fingerprint data willidentically match (or match with only one bit error) a correspondingstring in the reference database. Since these 32 bits are based on thestrengths of 32 narrow frequency bands between 300 Hz and 2000 Hz, thespectrum of the content can readily be tweaked to violate thisassumption, forcing a false-negative error. (E.g., notching out two ofthese narrow bands will force four bits of every 32 to a known state:two will go to zero—since these bands are lower in amplitude than thepreceding bands, and two will go to one—since the following bands arehigher in amplitude that these preceding, notched, bands). On average,half of these forced bits will be “wrong” (compared to the untweakedmusic), leading to two bit errors—violating the assumption on whichdatabase pruning is based.)

Attacks like the foregoing require a bit of effort. However, once anattacker makes the effort, the resulting hack can be spread quickly andwidely.

The exemplary fingerprinting technique noted above (which is understoodto be the basis for Gracenote's commercial implementation, MusicID,built from technology licensed from Philips) is not unique in beingvulnerable to various attacks. All fingerprinting techniques (includingthe recently announced MediaHedge, as well as CopySense and RepliCheck)are similarly believed to have vulnerabilities that can be exploited byhackers. (A quandary for potential adopters is that susceptibility ofdifferent techniques to different attacks has not been a focus ofacademic attention.)

It will be recognized that crowdsourcing can help mitigate thevulnerabilities and uncertainties that are inherent in fingerprintingsystems. Despite a “no-match” returned from the fingerprint-basedcontent identification system (based on its rote search of the databasefor a fingerprint that matches that of the altered content), thetechniques detailed herein allow human judgment to take a “second look.”Such techniques can identify content that has been altered to avoid itscorrect identification by fingerprint techniques. (Again, once suchidentification is made, corresponding information is desirably enteredinto the database to facilitate identification of the altered contentnext time.)

It will be recognized that the “crowdsourcing” methodologies detailedabove also have applicability to other tasks involved in thearrangements detailed in this specification, including all the documentsincorporated by reference.

Implementation of systems according to the present technology isstraightforward to artisans, e.g., using standard computer-, database-,software- and network-technology.

Although not particularly illustrated, it will be recognized that themethods described above can be implemented using general purpose (orspecial purpose) computers, e.g., comprising one or more CPUs,semiconductor memory, hard disks, networking connections, andinput-output devices, as are conventional in the art. Softwareinstructions for implementing the above-detailed methods can be storedon tangible media associated with such systems, e.g., disks andsemiconductor memories.

To provide a comprehensive disclosure without unduly lengthening thisspecification, applicants incorporate-by-reference the documentsreferenced in this disclosure. In addition to those noted elsewhere,these incorporated documents include application Ser. No. 10/979,770(now U.S. Pat. No. 7,314,162) and Ser. No. 11/132,724 (published asUS20050288952); published applications US20030052768, US20030099379.US20030115459, US20030216988, US20040059953, US20040064415,US20040153663, US20040189441, US20040205030, US20040213437,US20040230527, US20040245330, US20050039057, US20050132235,US20050154924, and US20050171851, and issued U.S. Pat. Nos. 5,679,938,5,679,940, 6,513,018, 6,597,775, 6,944,604, 6,965,889, and 6,968,328.

It is expressly contemplated that the technologies, features andanalytical methods detailed herein can be incorporated into themethods/systems detailed in such other documents. Moreover, thetechnologies, features, and analytical methods detailed in thosedocuments can be incorporated into the methods/systems detailed herein.(It will be recognized that the brief synopses of prior documentsprovided above naturally do not reflect all of the features found insuch disclosures.)

In view of the wide variety of embodiments to which the principles andfeatures discussed above can be applied, it should be apparent that thedetailed embodiments are illustrative only and should not be taken aslimiting the scope of the disclosed technology. Rather, I claim all suchmodifications as may come within the scope and spirit of the followingclaims and equivalents thereof.

I claim:
 1. A method comprising: generating fingerprint data from useruploaded visual or audio content, the user uploaded visual or audiocontent having been uploaded by a user to a first party's online videoservice for distribution from the first party's online video service tothe public; identifying plural items of reference content, from auniverse of reference content, that are a possible match of the useruploaded visual or audio content, said identifying being based on thefingerprint data and being performed by a computer system configured toperform such act; assessing a score for each of the identified pluralitems of reference content, each score indicating a likelycorrespondence of the user uploaded visual or audio content with arespective item of reference content; for items in the plural items ofreference content that fall within a predetermined score range, queuingfor review by a second party, different than the first party, the useruploaded visual or audio content item and data associated with the itemsin the plural items of reference content that fall within apredetermined score range; receiving assessment data from the secondparty, the assessment data indicating a result of a subjectivecomparison of the user uploaded visual or audio content and the items inthe plural items of reference content that fall within a predeterminedscore range; using the assessment data and a score assessed in said actof assessing in concluding that at least one item in the identifiedplural items of reference content corresponds with the user uploadedvisual or audio content; and limiting distribution of the user uploadedvisual or audio content from the first party's online video service tothe public based on the concluding.
 2. The method of claim 1 in whichthe user uploaded visual or audio content has been intentionally alteredto avoid fingerprint matching.
 3. The method of claim 2 in which theuser uploaded visual or audio content has been rotated.
 4. The method ofclaim 1 further comprising updating the universe of reference contentwith the user uploaded visual or audio content.
 5. The method of claim 1in which said limiting distribution comprises interrupting delivery to apublic user.
 6. The method of claim 5 further comprising providing anexplanatory message to the public user.
 7. The method of claim 1 inwhich the first party provides a graphical interface to allow access bythe second party for reviewing queued items.
 8. The method of claim 1 inwhich said queuing includes providing a second party a link to theplural items of reference content.
 9. The method of claim 1 in whichsaid generating fingerprint data comprises comprising analyzing atransform domain representation of the user uploaded visual or audiocontent.
 10. The method of claim 1 in which the data associated with theitems in the plural items of reference content that fall within apredetermined score range comprises at least a portion of the pluralitems of reference content.
 11. A computer system comprising: one ormore processors; a non-transitory computer-readable readable mediumstoring executable code, the code when executed by the one or moreprocessors causes the computer system to perform the following:generating fingerprint data from user uploaded visual or audio content,the user uploaded visual or audio content having been uploaded by a userto a first party's online video service for distribution from the firstparty's online video service to the public; identifying plural items ofreference content, from a universe of reference content, that are apossible match of the user uploaded visual or audio content, saididentifying being based on the fingerprint data and being performed by acomputer system configured to perform such act; assessing a score foreach of the identified plural items of reference content, each scoreindicating a likely correspondence of the user uploaded visual or audiocontent with a respective item of reference content; for items in theplural items of reference content that fall within a predetermined scorerange, queuing for review by a second party, different than the firstparty, the user uploaded visual or audio content item and dataassociated with the items in the plural items of reference content thatfall within a predetermined score range; receiving assessment data fromthe second party, the assessment data indicating a result of asubjective comparison of the user uploaded visual or audio content andthe items in the plural items of reference content that fall within apredetermined score range; using the assessment data and a scoreassessed in said assessing in concluding that at least one item in theidentified plural items of reference content corresponds with the useruploaded visual or audio content; and limiting distribution of the useruploaded visual or audio content from the first party's online videoservice to the public based on the concluding.
 12. The computer systemof claim 11 in which the user uploaded visual or audio content has beenintentionally altered to avoid fingerprint matching.
 13. The computersystem of claim 12 in which the user uploaded visual or audio contenthas been rotated.
 14. The computer system of claim 11 further comprisingcode that when executed by the one or more processors causes thecomputer system to update the universe of reference content with theuser uploaded visual or audio content.
 15. The computer system of claim11 in which the limiting distribution comprises interrupting delivery toa public user.
 16. The computer system of claim 15 further comprisingcode that when executed by the one or more processors causes thecomputer system to provide an explanatory message to the public user.17. The computer system of claim 11 further comprising code that whenexecuted by the one or more processors causes the computer system toprovide a graphical interface to allow access by the second party forreviewing queued items.
 18. The computer system of claim 11 in whichsaid queuing includes providing the second party a link to the pluralitems of reference content.
 19. The computer system of claim 11 thatfurther comprises code that when executed by the one or more processorscauses analyzing a transform domain representation of the user uploadedvisual or audio content.
 20. The computer system of claim 11 in whichthe data associated with the items in the plural items of referencecontent that fall within a predetermined score range comprises at leasta portion of the plural items of reference content.
 21. A computersystem comprising: an input to receive user uploaded audio-visualcontent, the user uploaded audio-visual content having been uploaded bya user to a first party's online video service for distribution from thefirst party's online video service to the public; and one or moreprocessors configured for: generating fingerprint data from datarepresenting audio or video data associated with the user uploadedaudio-visual content; identifying at least one item of referencecontent, from a universe of reference content, that is a possible matchof the user uploaded audio-visual content, the identifying being basedon the fingerprint data; determining a correspondence between the useruploaded audio-visual content and the at least one item of referencecontent; providing a computer interface for access by a second party,the second party being different than the first party; upon a conditionin which the correspondence falls within a predetermined range: queuinga comparison for review by the second party via the computer interface,the computer interface providing access to the user uploadedaudio-visual content and access to the at least one item of referencecontent, receiving assessment data from the second party, through thecomputer interface, the assessment data being associated with a resultof the comparison of the user uploaded audio-visual content and the atleast one item of reference content, and controlling distribution of theuser uploaded audio-visual content on the first party's online videoservice based at least in part on the assessment data, in which the useruploaded audio-visual content remains publically accessible pendingreceipt of the assessment data; and upon a condition in which thecorrespondence falls above the predetermined range: controllingdistribution of the user uploaded audio-visual content from the firstparty's online video service to the public per a rights holder'srequest.
 22. The computer system of claim 21 in which the user uploadedaudio-visual content has been altered to avoid fingerprint matchingprior to uploading.
 23. The computer system of claim 21 in which theuser uploaded audio-visual content has been rotated prior to uploading.24. The computer system of claim 21 in which the one or more processorsare configured to cause the computer system to update the universe ofreference content with the user uploaded audio-visual content.
 25. Thecomputer system of claim 21 in which the controlling distribution of theuser uploaded audio-visual content on the first party's online videoservice based at least in part on the assessment data comprisesinterrupting delivery to a public user.
 26. The computer system of claim25 in which the one or more processors are configured to cause thecomputer system to provide an explanatory message to the public user inconnection with the controlling distribution.
 27. The computer system ofclaim 21 in which the computer interface provides a graphical interfaceto allow access by the second party for reviewing queued items.
 28. Thecomputer system of claim 21 in which the computer interface providesaccess to the user uploaded audio-visual content and access to the atleast one item of reference content through links to content.
 29. Thecomputer system of claim 21 in which the one or more processors areconfigured to analyze a transform domain representation of the useruploaded audio-visual content, and generate the fingerprint data fromthe transform domain representation.
 30. The computer system of claim 21in which the one or more processors are configured to cause the computersystem to associate the fingerprint data with the universe of referencecontent.
 31. The computer system of claim 21 in which the possible matchcomprises a match that is less than conclusive.
 32. The computer systemof claim 21 in which the predetermined range comprises a range below aconclusive match and above a no-match condition.
 33. One or morecomputer servers associated with a first party's online videodistribution site, comprising: an input to receive user uploadedaudio-visual content, the user upload audio-visual content having beenuploaded by a user to the first party's online video distribution sitefor distribution from the first party's online video distribution siteto the public, in which the user uploaded audio-visual content has beenaltered relative to a corresponding version of the user uploadedaudio-visual content; one or more processors programmed for: i.generating fingerprint data from data representing audio or video dataassociated with the user uploaded audio-visual content; ii. identifyingreference content, from a universe of reference content, that is apossible match of the user uploaded audio-visual content, theidentifying being based at least on the generated fingerprint data; iii.determining a relative correspondence between the generated fingerprintdata and fingerprint data associated with the reference content; iv.providing a computer controlled interface associated with the firstparty's online video distribution site through which a result of manualhuman review of a queue generated by the first party's online videodistribution site is obtained, the queue for review by a second party,different than the first party; v. soliciting manual human reviewthrough the computer controlled interface upon a condition in which therelative correspondence indicates an unreliability, and providing accessto a copy of the user uploaded audio-visual content via the queue of thecomputer controlled interface, vi. in response to providing access tothe copy of the user uploaded audio-visual content via the queue of thecomputer controlled interface, obtaining manual human assessment datafrom the second party through the computer controlled interfaceassociated with the first party's online video distribution site, themanual human assessment data indicating a result of a comparison of thecopy of the user uploaded audio-visual content and data associated withthe reference content by one or more human reviewer(s), and vii.controlling distribution of the user uploaded audio-visual content fromthe first party's online video distribution site to the public based atleast in part on the manual human assessment data, in which the useruploaded audio-visual content remains publically accessible on the firstparty's online video distribution site pending receipt of the manualhuman assessment data; and viii. upon a condition in which the relativecorrespondence indicates reliability, controlling distribution of theuser uploaded audio-visual content from the first party's online videodistribution site to the public per a rights holder's request.
 34. Theone or more computer servers of claim 33 in which the determining arelative correspondence between the generated fingerprint data and thefingerprint data associated with the reference content utilizes ascoring such that the manual human review is solicited through thecomputer controlled interface upon a condition in which the scoringindicates the unreliability.
 35. The one or more computer servers ofclaim 33 in which the user uploaded audio-visual content has beenaltered to avoid fingerprint matching prior to uploading.
 36. The one ormore computer servers of claim 33 in which the user uploadedaudio-visual content has been rotated prior to uploading.
 37. The one ormore computer servers of claim 33 in which the one or more processorsare configured to cause said one or more computer servers to update theuniverse of reference content with the user uploaded audio-visualcontent.
 38. The one or more computer servers of claim 33 in which thecontrolling distribution of the user uploaded audio-visual content fromthe first party's online video distribution site to the public based onat least in part on the manual human assessment data comprisesinterrupting delivery to a public user.
 39. The one or more computerservers of claim 38 in which the one or more processors are configuredto provide an explanatory message to the public user in connection withthe controlling distribution of the user uploaded audio-visual contentfrom the first party's online video distribution site to the publicbased on at least in part on the manual human assessment data.
 40. Theone or more computer servers of claim 33 in which the computer interfaceprovides a graphical interface to allow access by the second party forreviewing queued items.
 41. The one or more computer servers of claim 33in which the computer interface provides access to the user uploadedaudio-visual content and access to the at least one item of referencecontent through links to content.
 42. The one or more computer serversof claim 33 in which the one or more processors are configured toanalyze a transform domain representation of the user uploadedaudio-visual content.
 43. The one or more computer servers of claim 33in which the one or more processors are configured to cause the one ormore computer servers to associate the generated fingerprint data withthe universe of reference content.
 44. The one or more computer serversof claim 33 in which the possible match comprises a match that is lessthan conclusive.
 45. The one or more computer servers of claim 33 inwhich the relative correspondence indicates an unreliability when thepossible match falls within a predetermined range, the predeterminedrange comprising a range below a conclusive match and above a no-matchcondition.
 46. The one or more computer servers of claim 33 in which thegenerated fingerprint data comprises first plural hashes and thefingerprint data associated with the reference content comprises secondplural hashes, in which the possible match is determined by a distancefunction comparing the first plural hashes with the second pluralhashes.
 47. The one or more computer servers of claim 33 in which theassessment data comprises subjective human judgment of one or more humanreviewer(s) pre-qualified as knowledgeable about the audio-visualcontent.
 48. The one or more computer servers of claim 33 in which thecontrolling distribution of the user uploaded audio-visual content fromthe first party's online video distribution site to the public based onat least in part on the manual human assessment data considers both themanual human assessment data and a result of the identifying referencecontent, but gives greater weight to the manual human assessment datarelative to the result of the identifying reference content.
 49. The oneor more computer servers of claim 33 in which the identifying referencecontent utilizes a Viterbi function.